RL Training-Inference Mismatch, Diagnostics, and Solution

Training-Inference Mismatch refers to a class of issues that the logprob of tokens mismatch between inference stage and training stage, causing training problems like reward dropping, etc. Results could be amplified in case of MoE training. This post briefly discusses related problems, diagnostics, and possible causes.